As applications (and other computer code) are developed, the applications, for example, are often tested, sometimes extensively, to identify and correct problematic code. In particular, developers may desire to test user interface (“UI”) code to improve experiences for users of the application. Frequently, such testing is performed by testing automated interactions with one or more user interface (“UI”) elements of windows that are displayed by an application, such as text, images, buttons, video displays, etc. Additionally, other automated interactions with UI elements may be performed in contexts other than testing. For example, in some contexts it may be desirable to record interactions performed by a user with an application, such as to record usage tutorials or to troubleshoot user issues.
However, using existing techniques, it can be difficult to obtain information about available UI elements in order to facilitate such testing. In particular, it may not be clear, in a given context, which UI elements are available, and/or being displayed at a particular time. Further, for those that are available, various properties of these UI elements, (such as whether they are being displayed, whether they are active, what text and/or image they are displaying, etc.) may not be readily available to a testing application seeking to perform interactions to test an application under test or by other applications or developers seeking automated interactions. In these properties may be used to validate that UI elements are displayed as expected and/or that objects related to the UI elements are operating as expected. In some techniques, a testing application may seek to obtain information about UI elements by directly accessing screen data, in effect “scraping” information from the screen. However, this can be an inexact technique, and may be particularly error-prone when applied to difficult-to-parse UI information, such as ideographic languages or images.
In some techniques, a testing application may directly access UI element information through programmatic techniques. However, in many devices and environments, windows, and the UI elements they contain, may be rendered by different rendering engines. These rendering engines may additionally be implemented in different languages or environments, such as Java, HTML, JavaScript, etc. Thus, a developer seeking to directly obtain information through programmatic techniques may find himself or herself using wildly different techniques to obtain UI element information from the different rendering engines.